Toward a New Generation of Virtual Humans for Interactive Experiences

and a civilian car. A seriously injured child lies on the ground, surrounded by a distraught woman and a medic from your team. You ask what happened and your sergeant reacts defensively. He casts an angry glance at the mother and says, “They rammed into us, sir. They just shot out from the side street and our driver couldn’t see them.” Before you can think, an urgent radio call breaks in: “This is Eagle1-6. Where are you guys? Things are heating up. We need you here now!” From the side street, a CNN camera team appears. What do you do now, lieutenant? Interactive virtual worlds provide a powerful medium for entertainment and experiential learning. Army lieutenants can gain valuable experience in decision-making in scenarios like the example above. Others can use the same technology for entertaining role-playing even if they never have to face such situations in real life. Similarly, students can learn about, say, ancient Greece by walking through its virtual streets, visiting its buildings, and interacting with its people. Scientists and science fiction fans alike can experience life in a colony on Mars long before the required infrastructure is in place. The range of worlds that people can explore and experience with virtual-world technology is unlimited, ranging from factual to fantasy and set in the past, present, or future. Our goal is to enrich such worlds with virtual humans—autonomous agents that support face-toface interaction with people in these environments in a variety of roles, such as the sergeant, medic, or even the distraught mother in the example above. Existing virtual worlds, such as military simulations and computer games, often incorporate virtual humans with varying degrees of intelligence. However, these characters’ability to interact with human users is usually very limited: Typically, users can shoot at them and they can shoot back. Those characters that support more collegial interactions, such as in children’s educational software, are usually very scripted and offer human users no ability to carry on a dialogue. In contrast, we envision virtual humans that cohabit virtual worlds with people and support face-to-face dialogues situated in those worlds, serving as guides, mentors, and teammates. Although our goals are ambitious, we argue here that many key building blocks are already in place. Early work on embodied conversational agents1 and animated pedagogical agents2 has laid the groundwork for face-to-face dialogues with users. Our prior work on Steve3,4 (see Figure 1) is particularly relevant. Steve is unique among interactive animated agents because it can collaborate with people in 3D virtual worlds as an instructor or teammate. Our goal with Steve is to integrate the latest advances from separate research communities into a single agent architecture. While we continue to add more sophistication, we have already implemented a core set of capabilities and applied it to the Army peacekeeping example described earlier. Virtual humans—